Knowledge Distillation for Closed-Source Language Models

Closed-source language models such as GPT-4 have achieved remarkable performance. Many recent studies focus on enhancing the capabilities of smaller models through knowledge distillation from closed-source language models. However, due to the incapability to directly access the weights, hidden states, and output distributions of these closed-source models, the distillation can only be performed by fine-tuning smaller models with data samples generated by closed-source language models, which constrains the effectiveness of knowledge distillation. In this paper, we propose to estimate the output distributions of closed-source language models within a Bayesian estimation framework, involving both prior and posterior estimation. The prior estimation aims to derive a prior distribution by utilizing the corpus generated by closed-source language models, while the posterior estimation employs a proxy model to update the prior distribution and derive a posterior distribution. By leveraging the estimated output distribution of closed-source language models, traditional knowledge distillation can be executed. Experimental results demonstrate that our method surpasses the performance of current models directly fine-tuned on data generated by closed-source language models

Fault Detection Based on Tracking Differentiator Applied on the Suspension System of Maglev Train

A fault detection method based on the optimized tracking differentiator is introduced. It is applied on the acceleration sensor of the suspension system of maglev train. It detects the fault of the acceleration sensor by comparing the acceleration integral signal with the speed signal obtained by the optimized tracking differentiator. This paper optimizes the control variable when the states locate within or beyond the two-step reachable region to improve the performance of the approximate linear discrete tracking differentiator. Fault-tolerant control has been conducted by feedback based on the speed signal acquired from the optimized tracking differentiator when the acceleration sensor fails. The simulation and experiment results show the practical usefulness of the presented method